Improving Test Suites via Generated Specifications

This thesis presents a specification-based technique for generating, augmenting, and minimizing test suites. The technique is automatic but assumes the existence of a test case generator. The technique dynamically induces specifications from test suite executions. Test suites can be generated by adding cases until the induced specification stops changing. The resulting test suites have better fault detection than suites of the same size with 100% branch coverage. Augmenting an existing test suite, such as a code-covering suite, also increases its fault detection. Minimizing test suites while holding the generated specification constant compares favorably to previously-known techniques. These positive results can be explained by two insights, which the thesis also justifies experimentally. First, given an a priori specification (an oracle), the specification coverage of a test suite compares the suite’s induced specification with the oracle. Experiments show that specification coverage is correlated with fault detection, even when test suite size and code coverage are held constant. Second, when tests are added at random to a suite, specification coverage increases rapidly, then levels off at a high value. Even without knowing the ideal specification that would be induced by all possible tests, it is possible to produce a specification very near that one. The thesis’s test suite generation and augmentation technique increases the specification coverage of the test suite, but without knowing the oracle specification and without examining the code. In addition to improving fault detection, the technique generates a specification close to the oracle, which has many benefits in itself. This technical report is a reformatted version of the author’s Masters thesis of the same title, published by MIT in May, 2002. He was advised by Michael Ernst, Assistant Professor of Electrical Engineering and Computer Science.